Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: jarobin Date: Feb 6, 2013 4:35pm
Forum: web Subject: will past crawls stay removed after removing robots.txt?

I've added robots.txt to remove past versions of my site from the waybackmachine. As I understand I now have to wait until the waybackmachine crawls my site again and then my site will be removed from the waybackmachine. Do I have to do anything extra to make that happen?

Now if I remove the robots.txt after that. Will the past versions stay removed permanently? That's what I want. I plan to give up my domain soon and don't want any of the past versions of the site to remain in the archive.

Thanks.

Reply to this post
Reply [edit]

Poster: jory2 Date: Feb 6, 2013 5:17pm
Forum: web Subject: Re: will past crawls stay removed after removing robots.txt?

"Now if I remove the robots.txt after that. Will the past versions stay removed permanently?"
without a court-order?, probably not.

Here's one link that illustrates my point is fine detail.
You may not like what you read but it's better to be informed than to be completely surprised.
http://nickleroy.com/how-to-build-links-with-free-expired-content

Good luck!

Reply to this post
Reply [edit]

Poster: jarobin Date: Feb 6, 2013 7:11pm
Forum: web Subject: Re: will past crawls stay removed after removing robots.txt?

For now I'm more interested in what happens when my domain expires, wayback machine tries to crawl my site and finds no robots.txt.

Will the previous versions of my site (that were previously 'excluded') become available again on archive.org or are they deleted or blocked or what exactly?

edit: I'm not jory2 or anyone else previously on here. I'm just asking a question. I have nothing against archive.org. :)

This post was modified by jarobin on 2013-02-07 03:11:02

Reply to this post
Reply [edit]

Poster: jory2 Date: Feb 6, 2013 7:39pm
Forum: web Subject: Re: will past crawls stay removed after removing robots.txt?

jarobin - Jeff Kaplan seems to be the Admin for the web "collection", or at least one of them.
He would be the right person to ask and to receive a direct answer from.

Reply to this post
Reply [edit]

Poster: Tae wong Date: Feb 11, 2013 9:58pm
Forum: web Subject: Re: will past crawls stay removed after removing robots.txt?

You will get an 400 Bad request error if your browser has a big address line.
Your browser sent an invalid request.

Reply to this post
Reply [edit]

Poster: PDpolice Date: Feb 6, 2013 6:38pm
Forum: web Subject: Re: will past crawls stay removed after removing robots.txt?

Posting to yourself is a bad sign Jory2. You are also beginning to have some difficulty with the words 'it' is' and 'in' no mater who you pretend to be. Take a deep breath and calm down.

As for your hatred of the Archive, have you ever put into words your reasons for it?